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PREFACE 


This  document  Is  the  final  technical  report  (CORL  A003)  for  the  Factors  In 
Software  Quality  Study*  contract  number  F030602-76-C-0417.  The  contract  was 
performed  In  support  of  the  U.S.  Air  Force  Electronic  Systems  Division's 
(ESD)  and  Rome  Air  Development  Center's  (RADC)  mission  to  provide  standards 
and  technical  guidance  to  software  acquisition  managers. 

The  report  consists  of  three  volumes,  as  follows: 

Volume  I Concept  and  Definitions  of  Software  Quality 

Volume  II  Metric  Data  Collection  and  Validation 

Volume  III  Preliminary  Handbook  on  Software  Quality  for  an 
Acquisition  Manager 

The  objective  of  the  study  was  to  establish  a concept  of  software  quality  and 
provide  an  Air  Force  acquisition  manager  with  a mechanism  to  quantitatively 
specify  and  measure  the  desired  level  of  quality  In  a software  product. 
Software  metrics  provide  the  mechanism  for  the  quantitative  specification  and 
measurement  of  quality. 


This  third  volume  Is  a preliminary  stand-alone  reference  document  to  be  used 
by  an  acquisition  manager  to  Implement  the  techniques  established  during  the 
study. 
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SECTION  1 
INTRODUCTION 


) J PURPOSE 

In  the  acqu1$tt:1o|n  of  a new  software  system,  a piajpr  problem  facing  a System 
Progriam  Office  (SPO)  Is  to  specify  the  requirements  to  the  software  developer 
and  then  to  determine  whether  those  requirements  are  being  satlsifed  as  the 
software  system  evolves.  The  parameters  of  the  specification  center  about  the 
technical  definition  of  the  application  and  the  function  of  the  software  within 
the  overall  system.  Following  this  specification,  a realistic  schedule  and 
costs  are  negotiated. 

There  has  been  no  mechanism  for  specifying  the  qualities  or  characteristics 
of  the  software  - qualities  such  as  reliability,  maintainability,  usability, 
testability,  and  portability.  The  Importance  these  software  qualities 
which  go  beyond  the  technical  mission  has  been  recognized  In  recent  years  as 
a necessary  concern  for  software  development  managers. 

This  recognition  has  come  about  because  of  the  many  Instances  In  which  the 
consequences  of  not  considering  software  quality  has  driven  total  project  costs 
and  schedule  well  beyond  Initial  estimates.  It  has  been  found  that  the  costs 
throughout  the  total  life  cycle  are  more  affected  by  the  characteristics  of  the 
software  system  than  by  the  mission-oriented  functions  performed  by  the  software 
system.  Large-scale  software  systems  have  sometimes  proven  untestable,  un- 
modi flable,  and  largely  unusable  by  operations  personnel  because  of  the 
characteristics  of  the  software. 

While  the  application  functions,  cost,  and  schedule  aspects  of  development  can 
be  objectively  defined,  measured,  and  assessed  throughout  the  development  of 
the  system,  the  quality  desired  has  historically  been  definable  only  In  sub- 
jective terms.  This  occurs  because  the  SPO  has  no  quantifl cable  criteria 
qgalnst  which  to  Judge  the  quality  of  the  software  until  he  begins  tp  use  the 
system  uqi^er  operation  conditions.  This  usually  leaves  the  SPO  with  only  two 
alternatives:  to  Incur  Increased  costs  or  to  back  off  from  the  requlrenionts 
Initially  dpslred  for  the  system. 
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The  objectives  of  this  handbook  are  (1)  to  describe  to  the  acquisition  manager  j 

what  the  software  qualities  or  characteristics  are  that  should  be  Included  In  | 

a specification,  (2)  provide  a mechanism  for  objectively  specifying  the 
software  quality  requirements,  and  (3)  Introduce  a methodology  for  measuring  * i 

i 

the  level  of  software  quality  achieved. 

1.2  SCOPE 

This  handbook  Is  based  on  the  results  of  a study  conducted  In  support  of  the 
U.S.  Air  Force  Electronic  Systems  Division's  (ESD)  and  Rome  Air  Development 
Center's  (RADC)  mission  to  provide  standards  and  technical  guidance  to  soft- 
ware acquisition  managers.  The  study  represented  an  Initial  conceptual  Inves- 
tigation of  the  factors  of  software  quality  with  a limited  sample  validation 
of  the  concept.  Further  reasearch  and  demonstrations  of  the  concept  are 
planned.  With  this  fact  In  mind,  three  approaches  are  described  for  both  | 

specifying  and  measuring  software  qualities.  The  approaches  are  presented 
In  order  of  Increasing  quantification.  For  both  specification  and  measure- 
l went,  the  first  two  approaches  described  are  Immediately  Implementable  and 

usable,  while  the  third  approach  Is  In  Its  conceptual  Infancy.  Further  exper- 
' lence  and  analysis  are  required  to  derive  the  generally  usable  quantified 

relationships  required  by  the  third  approach. 

1.3  RELATIONSHIP  OF  HANDBOOK  TO  QUALITY  ASSURANCE  FUNCTION 
The  techniques  described  In  this  handbook  are  envisioned  as  an  Integral  part  of 

I an  overall  quality  assurance  program.  Two  Important  facets  of  quality  assurance 

are  covered  In  the  handbook;  software  quality  specification  and  software  quality 
evaluation.  A review  of  MIL-STD  483  and  490  provides  Insight  Into  how  these 

’ j 

techniques  fit  Into  current  software  development  and  quality  assurance  practices. 

Appendix  I,  System  Specification,  Type  A,  of  MIL-STD  490  covers  characteristics 
I (paragraph  3.2)  of  the  system  which  should  be  described  In  the  specification. 

Characteristics  such  as  reliability,  maintainability,  availability,  and  Inter- 
I changeability  are  mentioned  but  are  oriented  toward  hardware  systems.  The 

I software  quality  factors  described  In  Section  2 of  this  handbook  should  be 

i Incorporated  In  this  section  of  a system  specification.  Appendix  VI,  Computer 
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Program  Development  Specifications,  of  NIL-STD  490  and  Appendix  VI « Computer 
Program  Configuration  Item  Specification,  of  MIL-STO  483  also  relate  to  the 
specification  of  software  In  Section  3,  Requirements.  It  Is  stated  that 
the  requirements  "shall  contain  performance  and  design  requirements"  and 
"specify  design  constraints  and  standards  necessary  to  assure  compatibility 
of  the  CPCI."  The  design  requirements  and  standards  should  also  Include  those 
features  that  enhance  software  quality*  Section  2 of  this  handbook  provides 
the  terminology  and  procedures  for  Including  requirements  for  software  quality 
In  the  specification. 

With  respect  to  the  measurement  of  software  quality.  Section  4 of  the  afore- 
mentioned appendices  of  MIL-STDs  483  and  490  cover  quality  assurance  provisions. 
Current  emphasis  In  these  sections  Is  on  functional  testing,  the  "formal 
verification  of  the  performance  of  the  CPCI  in  accordance  with  the  requirements." 
These  quality  assurance  provisions  should  also  Include  evaluation  of  the  soft- 
ware quality.  Section  3 of  this  handbook  describes  approaches  utilizing  software 
metrics  which  quantify  and  formalize  the  software  quality  evaluation  procedure. 
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1.4  BENEFITS  OF  APPROACH 

The  concepts  of  software  quality  metrics  expressed  in  this  handbook  contribute 
to  a more  disciplined,  engineering  approach  to  software  quality  assurance.  The 
software  acquisition  manager  is  provided  with  conceptually  simple,  easy  to 
use  procedures  for  specifying  required  quality  in  more  precise  terminology. 

The  specification  procedures  force  a life-cycle  view  at  the  initial  planning 
stages  of  a software  development.  An  acquisition  manager  essentially  performs 
a tradeoff  analysis  between  the  quality  factors,  cost,  and  schedule  in  estab- 
lishing the  requirements  of  the  system.  The  software  developer  is  forced  to 
address  how  they  plan  to  build  the  required  quality  into  the  software. 

Specific  software  quality  attributes  required  are  Independent  of  the  design 
and  implementation  techniques  used  by  the  software  developer.  Further 
insurance  or  confidence  is  gained  from  this  more  precise  description  of 
the  quality  requirements. 
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The  software  quality  metrics  provide  a more  formal,  consistent  means  of 
evaluating/inspecting  the  software  products  developed  during  a software  system 
development  effort.  They  also  provide  a means  of  tracking  the  progress  since 
they  are  applied  at  several  points  In  time  during  the  development.  The 
consistency  provides  a basis  for  comparison  between  projects  or  between 
different  data  collectors. 


The  metric  Indicator  concept  adds  an  additional  dimension  1n  that  It  provides 
a mechanism  for  pinpointing  dlfflclencies.  Evaluation  is  required  to 
determine  what  corrective  action,  if  any,  need  be  taken. 

Many  of  the  metrics  can  be  automatically  collected  enhancing  their  accuracy 
and  consistency.  The  metrics  are  applied  during  all  phases  with  increasingly  j 

greater  confidence  In  their  Indication  of  quality*  The  accumulation  of  the  j 

metric  data  and  the  application  of  these  analytical  techniques  enhance  the  ] 

understanding  of  the  software  development  process  and  the  characteristics  of  ] 

J 

good  design  and  Implementation.  j 
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The  cumulative  effect  of  these  procedures  and  techniques  is  that  a quanti- 
tative relationship  between  metrics  and  software  quality  ratings  can  be 
derived  and  used  to  predict  how  well  the  software  development  is  progressing 
toward  achieving  the  required  quality  In  the  resulting  software  product. 

This  ability  to  specify  software  quality  and  quantitatively  measure  the 
development  In  terms  of  the  quality  will  assist  the  acquisition  manager  and 
software  developer  In  producing  higher  quality  products. 

1.5  HANDBOOK  ORGANIZATION 

The  handbook  has  been  organized  as  a reference  document  for  an  acquisition 
manager,  describing  the  concepts  of  software  quality  and  Identifying  where 
more  detailed  Information  can  be  found. 
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The  first  section  provides  Introductory  Information,  definitions,  and  recom- 
mendations for  use  of  the  handbook. 

The  second  section  describes  three  approaches  to  specifying  software  quality. 
Each  description  Is  organized  In  the  following  manner: 

e General  discussion  of  approach 
e Steps  to  be  followed 

The  third  section  describes  three  approaches  to  measuring  software  quality. 
Each  approach  Is  described  using  the  same  format  of  the  second  section. 

The  three  approaches  In  each  section  are  presented  In  order  of  Increasing 
formality  and  quantification  of  the  relationship  between  the  metrics  and 
the  quality  factors.  The  approaches  are  titled  and  can  be  easily  Identified 
In  the  handbook  as  shown  In  Table  1-1. 

1.6  RELATIONSHIP  OF  HANDBOOK  TO  FACTORS  IN  SOFTWARE  QUALITY  FINAL  REPORT 
This  handbook  Is  based  on  and  utilizes  the  concepts  described  In  the  Factors 
In  Software  Quality  Final  Report,  Volumes  I and  II,  and  previous  research 
efforts  related  to  software  quality  referenced  In  that  report.  The  report 
is  the  result  of  work  performed  for  ESD  and  RADC  under  contract  number 
F030602-76-C-0417. 

It  Is  recommended  that  the  Factors  In  Software  Quality  Final  Report  be  read 
and  used  as  a supporting  reference  for  Implementing  the  concepts  described 
In  this  handbook. 

It  Is  also  recommended  that  the  metric  table  (Table  6.2.1)  contained  In  the 
first  volume  of  that  report  be  used  as  a data  collection  form  In  applying 
the  metrics. 


1.7  OEFINITIOMS 

The  following  definitions  are  provided  as  explanation  of  terarinology  used  In 
this  handbook: 


• Software:  the  programs  and  documentation  associated  with  and  resulting 
from  the  software  development  process. 

0 Quality:  a general  term  applicable  to  any  trait  or  characteristic, 
whether  Individual  or  generic;  a distinguishing  attribute  which 
Indicates  a degree  of  excellence  or  Identifies  the  basic  nature  of 
something. 

0 Factor:  a condition  or  characteristic  which  actively  contributes  to 
the  quality  of  the  software.  For  standardization  purposes,  all  factors 
will  be  related  to  a normalized  cost  to  either  perform  the  activity 
characterized  by  the  factor  or  to  operate  with  that  degree  of  quality. 
For  example,  maintainability  Is  the  effort  required  to  locate  and 
fix  an  error  In  an  operational  program.  This  effort  required  may  be 
expressed  In  units  such  as  time,  dollars,  or  manpower.  The  following 
rules  apply  to  the  set  of  software  quality  factors: 

- A condition  or  characteristic  which  contributes  to  software  quality 

- A user>related  characteristic 

- Related  to  cost  either  to  perform  the  activity  characterized  by 
the  function  or  to  operate  with  that  degree  of  quality 

- Relative  characteristic  between  software  products 

f I 

The  last  rule,  that  a factor  Is  a relative  characteristic  between 
software  products,  requires  a brief  explanation.  Figure  1-1  Illus- 
trates the  relationship  between  a factor  and  the  cost  to  achieve 
different  levels  of  that  quality  factor.  As  an  example,  assume  the 
curve  describes  the  cost  versus  level  of  quality  relationship  for  the 
factor  reliability.  A much  lower  level  of  reliability,  which  costs 
less  to  achieve,  may  be  as  acceptable  to  a management  Information 
system  (HIS)  acquisition  manager  as  a much  higher  level  Is  to  a 
command  and  control  (C^)  manager  due  to  the  nature  of  the  Individual 
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applications.  So,  while  the  C final  product  may  have  a higher  degree  of 
' reliability  according  to  our  measures.  It  may  be  no  more  acceptable  to 

the  user  than  the  MIS  system  with  Its  lower  reliability  Is  to  Its  user. 


Figure  1-1  Relationship  of  Software  Quality  to  Cost 

• Criteria;  attributes  of  the  software  or  software  production  process 
by  which  the  factors  can  be  Judged  and  defined.  The  following  rules 
apply  to  the  criteria: 

- Attributes  of  the  software  or  software  products  of  the  development 
process:  I.e.,  criteria  are  software-oriented  while  factors  are 
user-ori ented 

- May  display  a hierarchical  relationship  with  subcriteria 

- May  affect  more  than  one  factor 

e Metrics:  quantitative  measures  of  the  software  attributes  related  to 
the  quality  factors.  The  measures  may  be  objective  or  subjective.  The 
units  of  the  metrics  are  chosen  as  the  ratio  of  actual  occurrences 
to  the  possible  number  of  occurrences. 

• Normalization  Function;  a formal  mathematical  relationship  between 
a set  of  metrics  and  a rating  of  a quality  factor. 


i 
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1.8  RECOWCNDED  USE  OF  HANDBOOK 

It  Is  recoNiMnded  that  this  handbook  be  used  by  an  acquisition  aanager  as  a 
reference  during  the  preparation  of  a software  system  specification,  or  a request 
for  proposal  for  a software  system  development,  and  during  evaluation  activities 
of  the  software  development  phase. 

The  concept  of  quality  metrics  as  described  In  this  handbook  Involves  the 
application  of  the  metrics  via  already  existing  control  mechanisms,  I.e.,  reviews, 
status  reports,  documentation  delivered  during  the  development,  and  source  code. 
The  current  emphasis  of  these  controls  Is  to  evaluate  the  schedule  and  cost 
performance  and  to  determine  the  functional  correctness  of  the  software  being 
developed.  The  quality  metrics  are  applied  to  these  control  vehicles  to  provide 
an  Indication  of  the  quality  of  the  software  product  being  developed.  This 
concept  Is  Illustrated  In  Figure  1-2. 

The  metric  data  collection  (measuring)  can  be  done  by  the  contractor  or 
developer  and  reported  during  reviews  or  It  can  be  performed  by  the  acquisition 
manager's  quality  assurance  or  verification  and  valldatloh  personnel. 

The  order  1n  which  the  three  approaches  to  specifying  and  measuring  software 
quality  are  presented  not  only  corresponds  to  the  Increasing  formality  of  the 
relationship  between  quality  and  the  metrics  but  also  represents  an  Increasing 
manpower  requirement  for  Implementation  and  an  Increasing  requirement  for 
experience  on  the  part  of  the  SPO  with  the  concepts.  Based  on  these  facts,  it 
Is  recommended  that  the  concepts  be  Incrementally  phased  Into  an  acquisition 
manager's  operation.  The  first  approach  described,  for  both  specifying  and 
measuring  quality,  can  be  Implemented  with  a minimum  amount  of  effort  and  will 
yield  Innedlate  benefits  to  the  acquisition  manager.  The  second  approach  can 
then  be  phased  In,  requiring  additional  effort,  but  providing  higher  confidence 
In  the  Indication  of  the  level  of  quality  being  achieved  during  the  development 
process.  The  phased  approach  to  the  Introduction  of  the  quality  metrics  con- 
cepts allows  for  training,  familiarization,  and  experience  to  be  gained  while 
the  concepts  are  actually  used  and  benefits  realized.  The  third  approach  will 
become  more  viable  as  experience  and  historical  data  Is  accumulated  using  the 
first  two  approaches.  The  mathematical  relationships  required  by  the  third 
approach  are  derived  from  the  historical  data. 
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SECTION  2 

SPECIFYING  SOFTWARE  QUALITY 
2.1  CONCEPT  OF  FACTORS  IN  SOFTWARE  QWlin 

An  acquisition  manager's  Involvement  with  a software  product  can  be  categorized 
In  terms  of  three  distinct  activities  as  follows: 


V 


SOFTWARE  PRODUCT  ACTIVITIES 


e Product  Operation 
e Product  Revision 
e Product  Transition 


Specific  qualities  or  characteristics  (quality  factors)  of  the  software  product 
are  related  to  these  activities  as  shown  In  Figure  2-1.  the  questions  In 
parentheses  provide  the  relevancy  or  Interpretation  of  the  factors  to  an 
acquisition  manager. 

Thus,  with  respect  to  the  operation  of  a software  system,  ary  acquisition  manager's 
concern  for  quality  Is  In  terms  of  Its  correctness,  reliability*  efficiency. 
Integrity,  and  usability.  Over  the  life  cycle  of  a system,  revisions  to  the 
system  may  be  necessary  due  to  problems  or  changing  requirements.  The 
acquisition  manager  Is  therefore  concerned  with  the  maintainability* 
flexibility,  and  testability  of  the  software.  Longer  range  considerations 
may  Involve  moving  the  software  to  another  harA»are  system.  Interfacing  It 
with  another  system,  or  developing  newer  versions  of  the  system.  The 
related  quality  concerns  are  for  portability.  Interoperability,  and  reusability. 

The  definitions  of  these  eleven  quality  factors  are  In  Table  2-1. 


All  of  the  quality  factors  should  be  considered  In  the  Initial  specification 
for  the  software.  The  first  approach  (paragraph  2.2)  describes  the  procedure 
for  considering  these  quality  factors.  The  following  paragraphs  (2.3.  2.4) 
describe  progressively  more  detailed  approaches  to  specifying  software  quality. 
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COWttCTIIESS 

ifFisiaa 

IWTEWITY 

tsaftmix 

miMTAlNAUllTY 

TISTABILITY 

aEXIBIllTY 

PORTABILITY 

t^USABILITY 


Extent  to  Mhlch  « prograia  satisfies  Its  specifications 
and  fulfills  the  user's  Mission  objectives. 

Extent  to  which  a program  can  be  expected  to  perform 
Its  Intended  function  with  required  precision. 

The  aMount  of  computing  resources  and  code  required  by 
a program  to  perform  a function. 

Extent  to  which  access  to  software  or  data  by 
unauthorized  persons  can  be  controlled. 

Effort  required  to  learn,  operate,  prepare  Input,  and 
Interpret  output  of  a program. 

Effort  required  to  locate  and  fix  an  error  In  an 
operational  program. 

Effort  required  to  test  a program  to  Insure  It  performs 
Its  Intended  function. 

Effbrt  required  to  modify  an  operational  program. 

Effort  required  to  transfer  a program  from  one  hardware 
configuration  and/or  software  system  environment  to 
another. 

Extent  to  which  a program  can  be  used  In  other 
applications  - related  to  the  packaging  and  scope  of  the 
functions  that  programs  perform. 


Effort  required  to  couple  one  system  with  another. 


J 

I 


j 
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2.2  SPECIFICATIONS  USIN6  SOFTWARE  QUALITY  FACTORS 

2.2.1  DISCUSSION  OF  FIRST  APPROACH 

This  first  approach  to  specifying  software  quality  uses  the  conceptualization 
of  factors  In  software  quality  described  In  the  previous  paragraph  as  the 
basic  MechanlsM  for  the  acquisition  manager  to  Identify  requirements  for  quality 
In  a software  product  which  have  complete  life-cycle  Implications.  For 
example.  If  the  SPO  Is  sponsoring  the  development  of  a system  in  an  environ- 
ment In  which  there  Is  a high  rate  of  technical  breakthroughs  In  hardware 
design,  portability  should  take  on  an  added  significance.  If  the  expected 
life  cycle  of  the  system  Is  long,  maintainability  becomes  a cost-critical 
consideration.  If  the  application  Is  an  experimental  system  where  the  software 
specifications  will  have  a high  rate  of  change,  flexibility  In  the  software 
product  Is  highly  desirable.  If  the  functions  of  the  system  are  expected  to 
be  required  for  a long  time,  while  the  system  Itself  may  change  considerably, 
reusability  Is  of  prime  Importance  In  those  modules  which  Implement  the 
major  functions  of  the  system.  With  the  advent  of  more  computer  networks 
and  communication  capabilities,  more  systems  are  being  required  to  Interface 
with  other  systems  and  the  concept  of  Interoperability  Is  extremely  Important. 
All  of  these  considerations  can  be  accommodated  In  the  framework  derived. 

2.2.2  STEPS  TO  BE  FOLLOWED 

1.  In  preparing  a request  for  proposal  (RFP)  or  system  requirements 
specification  (SRS),  the  acquisition  manager  should  Identify  and 
assign  priorities  to  the  critical  quality  factors. 

Each  software  system  Is  unique  In  Its  software  quality  requirements. 
There  Is  no  specific  categorization  of  applications  which  can  be 
related  to  certain  levels  of  quality.  There  are  certain  basic  system 
characteristics  which  effect  the  quality  requirements.  Each  system 
must  be  evaluated  for  Its  fundamental  characteristics.  These 
fundamental  characteristics  and  the  related  quality  factors  are 
Identified  In  Table  2-2. 
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These  basic  characteristics  should  be  taken  Into  account  when  the 
critical  quality  factors  are  Identified. 

In  considering  all  of  the  quality  factors,  the  life-cycle  Implications 
of  the  system  are  considered.  Table  2-3  Identifies  the  life-cycle 
Implications  (Impact  of  poor  quality)  of  the  quality  factors  and  should 
be  used  as  an  Input  In  Identifying  the  relative  Importance  of  the 
quality  factors. 

During  the  process  of  Identifying  the  Importance  of  the  quality 
factors,  the  tradeoffs  between  certain  quality  factors  should  be 
recognized.  Table  2-4  should  be  utilized  as  a guide  for  determining 
the  conflicts  In  quality  requirements.  A few  examples  are  provided 
to  Illustrate  how  the  table  Is  Interpreted: 

Maintainability  vs  Efficiency  - optimized  code.  Incorporating  Intricate 


Maintainability  vs  Efficiency  - optimized  code.  Incorporating  Intricate 
coding  techniques  and  direct  code,  always  provides  problems  to  the 
maintalner.  Using  modularity.  Instrumentation,  and  well  commented 
high-level  code  to  Increase  the  maintainability  of  a system  usually 
Increases  the  overhead,  resulting  In  less  efficient  operation. 

Integrity  vs  Efficiency  - the  additional  code  and  processing  required 
to  control  the  ^^ccess  of  the  software  or  data  usually  lengthen  run  time 
and  require  additional  storage. 

Interoperability  vs  Integrity  - coupled  systems  allow  for  mere  avenues 
of  access  and  different  users  who  can  access  the  system.  The  potential 
for  accidental  access  of  sensitive  data  Is  Increased  as  well  as  the 
opportunities  for  deliberate  access.  Often,  coupled  systems  share 
data  or  software,  which  compounds  the  security  problems  as  well. 


e 2-3  The  Impact  of  not  Specifying  or  Measuring  Software  Quality  Factors 


It  should  be  recognized  this  table  only  provides  general  guidelines  and 
further  analyses  along  these  lines  should  be  made  for  specific  cases. 

1 It  Is  Important  to  note  that  If  a high  level  of  quality  Is  required  for 

conflicting  factors,  the  cost  to  achieve  the  requirements  my  be  very 
high. 

2.  Once  the  critical  quality  factors  have  been  Identified  and  priorities 
■ assigned,  they  should  be  Included  In  the  RFP  or  SRS  with  definitions  from 

paragraph  2.1,  and  the  developer  required  to  comment  on  how  the  soft- 
ware will  be  developed  to  exhibit  the  qualities  specified. 

3.  Wherever  possible,  as  much  detailed  explanation  should  be  Included 
with  the  definition  for  each  quality  factor.  For  example.  If 
portability  Is  a major  concern  to  an  acquisition  manager,  as  precise 
a description  as  possible  should  be  Included  as  to  the  types  of 
environments  to  which  the  system  might  be  transported. 

2.3  IDENTIFICATION  OF  CRITICAL  SOFTWARE  AHRIBUTES 

2.3.1  DISCUSSION  OF  SECOND  APPROACH 

This  approach  Involves  a refinement  to  the  first  approach  described  In 
paragraph  2.2.  Each  quality  factor  Is  further  defined  by  criteria  which  are 
the  software  attributes  whose  presence  In  the  software  enhances  the  characteristic 
ri'presented  by  the  quality  factor. 

The  criteria  Identified  in  the  Factors  In  Software  Quality  Final  Report  are 
shown  In  Figure  2-2,  Indicating  which  quality  factors  they  significantly 
Impact,  and  are  defined  In  Table  2-5. 

These  criteria  are  dsed  in  this  approach  to  further  define  the  quality 
requirements. 

2.3.2  STEPS  TO  BE  FOLLOWED 

1.  Having  Identified  the  critical  quality  factors,  the  acquisition 
manager  then  Identifies  the  related  critical  software  attributes 
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Figure  2-2  Relationship  of  Criteria  to  Software  Quality  Factors 
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Figure  2-2  Relationship  of  Criteria  to  SoftMare  Quality  Factors  (continued) 
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CRITERION 

DEFINITION 

RELATED 

FACTORS 

TRACEABILITY 

Those  attributes  of  the  software  that  provide 
a thread  from  the  requirements  to  the  Imple- 
mentation with  respect  to  the  specific 
development  and  operational  environment. 

Correctness 

1 

COMPLETENESS 

Those  attributes  of  the  software  that 
provide  full  Implementation  of  the  functions 
required. 

Correctness 

CONSISTENCY 

Those  attributes  of  the  software  that 
provide  uniform  design  and  Implementation 
techniques  and  notation. 

Correctness 

Reliability 

Maintainability 

ACCURACY 

Those  attributes  of  the  software  that 
provide  the  required  precision  in  calcula- 
tions and  outputs. 

Reliability 

ERROR  TOLERANCE 

Those  attributes  of  the  software  that 
provide  continuity  of  operation  under 
nonnomlnal  conditions. 

Reliability 

SIMPLICITY 

Those  attributes  of  the  software  that 
provide  Implementation  of  functions  in  the 
most  understandable  manner.  (Usually 
avoidance  of  practices  which  Increase 
complexity.) 

Reliability 

Maintainability 

Testability 

MODULARITY 

Those  attributes  of  the  software  that 
provide  a structure  of  highly  Independent 
modules. 

Maintainability 

Flexibility 

Testability 

Portability 

Reusabi 1 1 ty 

Interoperability 

GENERALITY 

Those  attributes  of  the  software  that 
provide  breadth  to  the  functions  performed. 

Flexibility 

Reusability 

EXPANDABILITY 

Those  attributes  of  the  software  that 
provide  for  expansion  of  data  storage 
requirements  or  computational  functions. 

Flexibility 

INSTRUMENTATION 

Those  attributes  of  the  software  that 
provide  for  the  measurement  of  usage  or 
Identification  of  errors. 

Testability 

SELF- 

DESCRIPTIVENESS 

Those  attributes  of  the  software  that 
provide  explanation  of  the  Implementation 
of  a function. 

Flexibility 

Maintainability 

Testability 

Portability 

Reusability 

Table  2-5  Criteria  Definitions  for  Software  Quality  Factors  (continued) 


CRITERION 

— 

DEFINITION 

RELATED 

FACTORS 

EXECUTION 

EFFICIENCY 

Those  attributes  of  the  software  that 
provide  for  minimum  processing  time. 

Efficiency 

STORAGE 

EFFICIENCY 

Those  attributes  of  the  software  that 
provide  for  minimum  storage  requirements 
during  operation. 

Efficiency 

ACCtiiS  CONTROL 

Those  attributes  of  the  software  that 
provide  for  control  of  the  access  of 
software  and  data. 

Integrity 

ACCESS  AUDIT 

Those  attributes  of  the  software  that 
provide  for  an  audit  of  the  access  of 
software  and  data. 

Integrity 

OPERABILITY 

Those  attributes  of  the  software  that 
determine  operation  and  procedures  con- 
cerned with  the  operation  of  the  software. 

Usability 

TRAINING 

Those  attributes  of  the  software  that 
provide  transition  from  current  operation 
or  Initial  familiarization. 

Usability 

COMMUNICATIVENESS 

Those  attributes  of  the  software  that 
provide  useful  Inputs  and  outputs  which 
can  be  assimilated. 

Usability 

SOFTWARE  SYSTEM 
INDEPENDENCE 

Those  attributes  of  the  software  that 
determine  Its  dependency  on  the  software 
environment  (operating  systems,  utilities. 
Input/output  routines,  etc.) 

Portability 

Reusability 

MACHINE 

INDEPENDENCE 

Those  attributes  of  the  software  that 
determine  Its  dependency  on  the  hardware 
system. 

Portability 

Reusability 

COMMUNICATIONS 

COMMONALITY 

Those  attributes  of  the  software  that 
provide  the  use  of  standard  protocols 
and  Interface  routines. 

Interoperability 

DATA  COMMONALITY 

Those  attributes  of  the  software  that 
provide  the  use  of  standard  data  repre- 
sentations. 

Interoperability 

CONCISENESS  Those  attributes  of  the  software  that  Maintainability 

provide  for  Implementation  of  a function 
with  a minimum  amount  of  code. 
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to  stress  the  Importance  of  maintainability*  the  following  software 
attributes  would  be  Identified  as  required  In  the  RP  or  SRS: 
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Consistency  • j 

Simplicity 

Conciseness  i 

• i 

Nodularity 
Sel f-Descr 1 pti veness 

The  precise  definitions  of  these  attributes  would  be  Included  In 
the  RP  or  SRS. 

2.  The  acquisition  manager  should  evaluate  the  software  developer's 
plan  to  provide  the  required  software  attributes  for  each  quality 
factor. 

2.4  QUANTIFICATION  OF  SOFTWARE  QUALITY 

2.4.1  DISCUSSION  OF  THIRD  APPROACH 

This  approach  requires  precise  statements  of  the  level  of  quality  required  of 
the  software.  Currently*  the  underlying  mathematical  relationships  which  allow 
measurement  at  this  level  of  precision  do  not  exist  for  general  use.  The 
procedures  for  developing  those  relationships  are  documented  In  the  Factors 
In  Software  Quality  Final  Report.  The  mechanism  for  making  the  precise 
'Statement  for  wy  quality  factor  Is  a rating  of  that  factor.  Hie  ratings 
are  explained  In  Table  2-6. 

2.4.2  STEPS  TO  BE  FOLLOWED 

1.  After  Identification  of  the  critical  quality  factors,  specific 

performance  levels  or  ratings  required  for  each  factor  should  be  ' 

specified.  For  example*  a rating  for  amlntalnablllty  might  be 
that  the  average  time  to  fix  a problem  should  be  five  man-days  or  * 

that  90X  of  the  problem  fixes  should  take  less  than  six  man-days. 

This  rating  would  be  specified  In  the  RP.  To  comply  with  this 
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Table  2-6  Problem  Report  and  Man-Pomer  Expenditure  Categorization 


CATEGORY  BY 
QUM.ITY  FACTOR 

EXPLANATION 

e CORRECTNESS 

The  function  which  the  software  Is  to  perform  Is 
Incorrect.  The  rating  Is  In  terns  of  effort  required 
to  fix. 

e RELIABILITY 

• 1 

The  software  does  not  function  as  expected.  The 
rating  Is  In  terms  of  effort  required  to  fix. 

e EFFICIENCY 

The  software  does  not  Meet  perfonsance  (speed,  stor- 
age) requirements.  The  rating  Is  In  terms  of  effort 
required  to  fix. 

0 INTEGRITY 

The  software  does  not  provide  required  security. 

The  rating  Is  In  terms  of  effort  required  to  fix. 

e USABILITY 

There  Is  a problem  related  to  operation  of  the  soft- 
ware, the  user  Interface,  or  the  Input/output.  The 
rating  Is  In  terms  of  effort  requlr^  to  fix. 

e MAINTAINABILITY 

The  rating  Is  In  terms  of  effort  required  to  correct 
any  of  the  above  problems. 

e aEXIBILITY 

The  rating  Is  In  terms  of  effort  required  to  make  a 
modification  due  to  a change  In  specifications. 

e TESTABILITY 

The  rating  Is  in  terms  of  effort  required  to  test 
changes  or  fixes. 

e REUSABILin 

The  rating  Is  In  terns  of  effort  required  to  use 
software  in  a different  application. 

e PORTABILITY 

The  rating  Is  In  terms  of  effort  requf^  to  convert 
the  software  to  operate  In  a different  environment. 

e INTEROPERABILin 

The  rating  Is  In  terms  of  effort  required  to  couple 
the  system  to  another  system. 

specification,  tha  softmere  Mould  have  to  exhibit  characteristics  which 
when  present,  give  an  Indication  that  the  software  will  perform  to 
this  rating.  These  characteristics  are  measured  by  metrics.  The 
measurements  are  Inserted  In  a mathematical  relationship  and  a 
predicted  rating  Is  obtained. 

2.  The  specific  metrics  should  be  Identified  which  will  be  applied  to 
various  software  products  of  the  development  phase  to  provide  an 
Indication  of  the  progress  toward  achieving  the  required  level  of 
quality.  These  metrics  will  be  discussed  further  In  the  next  section 
and  are  defined  In  Section  6 of  the  Factors  In  Software  Quality  Final 


SECTION  3 

MEASURING  SOFTWARE  QUALITY 


3.1  THE  CONCEPT  OF  QUALITY  METRICS 

Figure  3-1  Illustrates  the  concept  of  applying  Retries  during  the  development 
of  a software  system.  The  metrics  are  quantitative  Measures  of  the  software 
attributes  (criteria  Identified  In  paragraph  2.3)  whIcN  arc  necessary  to  realize 
certain  characteristics  (quality  factors)  In  the  software.  The  Metrics 
provide  an  Indication  of  the  progression  toward  the  achleviMent  of  high  quality 
end  products.  Specific  acceptance  tests  can  be  oriented  toward  evaluating  the 
levels  of  quality  achieved  but  these  testing  strategies  are  not  within  the 
scope  of  this  handbook. 


As  previously  mentioned,  the  metrics  have  been  developed  to  be  applied  to 
products  currently  provided  during  a software  development.  They  may  be  applied 
either  by  acquisition  manager  personnel  to  delivered  products,  by  contractor 
personnel  and  reported  In  sunmary  format  to  the  acquisition  manager  during 
reviews,  or  by  contractor  personnel  as  part  of  their  own  quality  assurance 
program. 


The  metrics  were  developed  so  as  not  to  restrict  or  Interfere  with  the  manage- 
ment and  development  methodologies  and  techniques  of  the  developer. 


The  metrics  are  listed  In  Table  6.2-1  of  the  Factors  In  Software  Quality  Final 
Report  with  definitions  following  that  table.  Their  application  to  software 
products  Is  described  In  Appendix  D of  that  report.  Typical  automated  tools 
available  In  software  development  environments  which  assist  In  the  metric 
data  collection  are  Identified  In  Section  8 of  that  report  also. 

For  Illustration,  some  examples  of  metrics  are  provided  In  Table  3-1.  The 
complexity  measure  Is  calculated  from  a design  chart  and  from  source  code 
utilizing  path  flow  analysis  and  data  set/use  Information.  The  effectiveness 
of  comments  measure  Is  a quantitative  measure  based  on  objective  guidelines 
for  Inspecting  the  source  code  for  the  existence  or  absence  of  comments  at 
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METRIC 

MNEMONIC 

CRITERION 

RELATED  QUALin 
FACTORS 

Data  and  Control 
Flow  Conplexity 
Neasuro 

SI. 3 

Simplicity 

Reliability 

Maintainability 

Testability 

Effectiveness 
of  Comnents 
Measure 

S0.2 
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Flexibility 

Maintainability 

Reusability 

Portability 

Machine 

Independence 

Measure 

MI.l 

Machine 

Independence 

Portability 

Reusability 

Completeness 

Checklist 

CP.1 

Completeness 

Correctness 

specific  locations  such  as  prologue  conments  with  certain  Information, 
branching  statements,  machine  dependent  code,  declarative  statements,  and 
so  forth.  The  machine  Independence  measure  Is  a compilation  of  a number  of 
measurements  which  Indicate  the  degree  of  Independence  of  the  design  and 
cede.  The  completeness  checklist  measures  attributes  that  should  be  Included 
In  specifications,  design  dociaeents,  and  the  code,  which.  If  missing.  Increase 
the  probability  that  the  final  product  will  be  functionally  Incomplete. 

The  fellewing  paragraphs  describe  Increasingly  more  detailed  approaches 
to  utilizing  these  metrics  to  provide  an  Indication  of  the  quality  of  the 

software  being  developed. 


3.2  mmL  msytcTioH  of  somomE  products  using  metrics 

3.2.1  OISCUbSIQN  OF  APPROACH 

The  first  level  of  measuring  software  quality  Involves  applying  the  metrics 
to  software  products  as  they  are  produced.  Different  sets  of  metrics  are 
applicable  to  products  produced  during  the  requirements  analysis,  design, 
and  ceding  phases  of  development.  The  use  of  the  metrics  In  this  manner 
Insures  a formal  and  consistent  review  of  each  of  the  software  products. 

3.2.2  STEPS  TO  BE  FOLLOWED 

1.  The  subset  of  metrics  vrtilch  relate  to  the  Identified  critical  quality 
factors  and  software  attributes  and  are  applicable  to  the  phase  of 
development  should  be  applied  to  the  available  software  products. 

For  example,  during  the  design  phase,  metrics  could  be  applied  to 
design  specifications,  interface  control  documents,  test  plans, 
minutes  and  materials  prepared  for  reviews,  and  so  on. 


2.  A subjective  evaluation  of  how  well  the  software  Is  being  developed 
with  respect  to  the  specific  quality  factors  can  be  made  based  on  the 
Inspection  of  the  software  products  using  the  metrics. 


3.3 


3.3.1  DISCUSSION  OF  APPROACH 

The  second  approach  utilises  experience  gained  through  the  application  of 
metrics  and  the  accumulation  of  historical  information  to  take  advantage 
of  the  quantitative  nature  of  the  metrics.  The  values  of  the  measurements 
are  es  ind1(:ator$  for  evaluation  of  the  progress  toward  a high  quality 

[ I product. 

' 


r 
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3.3.P  STEPS  TO  BE  FOLEOWED 

1.  After  the  metrics  are  applied  to  the  available  software  products, 
the  values  are  obtained  and  evaluated.  If  particular  modules  receive 
low  metric  scores,  they  can  be  individually  evaluated  for  potential 
problems.  If  low  metric  scores  are  realized  across  the  system,  an 
evaluation  should  be  made  to  identify  the  cause.  It  may  be  that  a 
design  or  implementation  technique  used  widely  by  the  development 
team  is  the  cause.  Corrective  action  such  as  the  enforcement  of  a 
development  standard  can  then  be  introduced. 

2.  Further  analysis  can  be  conducted,  An  examination  of  the  metric 
scores  for  each  module  in  a system  will  reveal  which  metrics  vary 
widely.  Further  examination  will  reveal  if  this  variation  correlates 
with  the  number  of  problem  reports  or  with  historical  variances  in 
performance.  This  sensitivity  analysis  identifies  characteristics 

of  the  software,  represented  by  the  metrics,  which  are  critical  to 
the  quality  of  the  product.  Quality  assurance  personnel  should 
place  increased  emphasis  on  these  aspects  of  the  software  product. 

3.  Threshold  values  may  be  established  below  which  certain  actions 
would  be  required.  A simple  example  is  the  percent  of  comments 
per  line  of  source  code.  Certainly  code  which  exhibits  only  IX 

or  2X  measurements  for  this  metric  would  be  Identified  for  correc- 
tive action.  It  may  be  that  10X  to  201  is  a more  industry-wide 
acceptable  level. 
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3.4  FORMAL  RELATIONSHIP  OF  METRICS  TO  QUALITY  FACTORS 


3.4.1  DISCUSSION  OF  APPROACH 

This  approach  Is  the  nost  detailed  of  the  three  approaches  to  measuring  soft- 
ware quality.  The  underlying  mathematical  foundations  to  the  derivation  of 
the  relationships  are  described  in  Section  7 and  Appendix  C of  the  Factors  In 
Software  Quality  Final  Report.  Basically,  the  measurements  (m^)  for  a 
subset  of  metrics  are  applied  to  the  software  products  of  a specific  phase 
(0)  In  the  software  development.  When  Inserted  Into  the  corresponding  equa- 
tion (normalization  function)  a rating  for  a particular  quality  factor  (r^) 
can  be  predicted  as  shown  below: 


f^  (m^,  m2 m,^)  = rp 


Currently,  generally  applicable  predictive  equations  are  not  available. 
Specific  normalization  functions  were  developed  during  the  study  which 
resulted  In  this  handbook.  They  are  based  on  a limited  sample  and  are  not 
recommended  for  general  use.  The  procedure  for  the  derivation  of  equations 
which  would  be  very  useful  In  a partcllar  development  environment  Is 
described  In  detail  within  the  report. 

3.4.2  STEPS  TO  BE  FOLLOWED 

1.  To  Illustrate  the  procedures  Involved  In  this  approach,  a normali- 
zation function  for  th»;  quality  factor  flexibility  developed 
during  the  Factors  In  Software  Quality  study  will  be  used.  The 
normalization  function,  applicable  during  the  design  phase, 
relates  measures  of  modular  Implementation  (M0.2p)  to  the  flexibility 
of  the  .software.  The  predicted  rating  of  flexibility  Is  In  terms 
of  the  average  time  to  Implement  a change  In  specifications.  The 
normalization  function  Is  shown  In  Figure  3-2. 


MO  2 

The  measurements  associated  with  are  taken  from  design  documents  and 

reveal  If  Input,  output,  and  processing  functions  are  mixed  In  the  same 
module.  If  application  and  machine-dependent  functions  are  mixed  In  the  same 
module,  and  If  processing  Is  data  volume  or  data  value  limited. 


! i 
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As  an  example,  assume  the  measurements  were  applied  during  the  design  phase 
of  a project  and  a metric  value  of  .65  was  measured.  Inserting  this  value 
In  the  normalization  function: 


results  In  a predicted  rating  for  flexibility  of  .33  (Identified  by  point  A 
In  Figure  3-2).  If  the  acquisition  manager  had  specified  a rating  of  .2 
(1/5  average  man-days  to  change),  which  Is  Identified  by  point  B In  Figure  3-2, 
he  has  an  Indication  that  the  software  development  Is  progressing  well  with 
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respect  to  this  desired  quality.  By  analyzing  the  variance  associated  with 
this  normalization  function.  It  Is  shown  in  Figure  3-3  that  the  acquisition 
manager  has  an  86%  level  of  confidence  that  the  flexibility  of  his  system 
will  be  better  than  his  specified  rating. 

MEAN  = .33 


(SPECIFIED  RATING)  .2 


MEAN  - .33  (PREDICTED  RATING) 

STANDARD  DEVIATION  * .12  (STANDARD  ERROR  OF  ESTIMATE) 
LEVEL  OF  CONFIDENCE  = Pr  {x  i.2}  « .86  (SHADED  AREA) 

Figure  3-3  Determination  of  Level  of  Confidence 


The  comparison  of  the  predicted  rating  with  the  specified  rating 
provides  a more  quantitative  Indication,  with  an  associated  level  of 
confidence,  of  how  well  the  software  development  Is  progressing 
toward  achieving  the  specified  levels  of  quality.  Corrective 
action  based  on  further  analysis  would  be  In  order  If  the  predicted 
rating  was  lower  than  the  specified  rating. 
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